Method and device for processing adaptive media service, encoder and decoder

ABSTRACT

According to embodiments of the present disclosure, a method for processing an adaptive media service at an encoder includes a first acquisition step of acquiring a first data stream including first image encoding data obtained by encoding a first image sequence, a second acquisition step of acquiring at least one second data stream, each second data steam including second image encoding data obtained by encoding a second image sequence and a target optimization parameter corresponding to the second image encoding data, a first selection step of selecting one data stream from a first data stream set in accordance with a condition of the receiver, the first data stream set at least including the first data stream and the at least one second data stream, and a first transmission step of transmitting the selected data stream to the receiver.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is the U.S. national phase of PCT Application No.PCT/CN2016/071426 filed on Jan. 20, 2016, which claims priority toChinese Patent Application No. 201510552867.2 filed on Sep. 1, 2015, thedisclosures of which are incorporated in their entirety by referenceherein.

TECHNICAL FIELD

The present disclosure relates to the field of multimedia service, inparticular to a method and a device for processing an adaptive mediaservice, an encoder and a decoder.

BACKGROUND

An increase in multimedia services (e.g., streaming media televisionsand video conference services) is an important driving factor for theinnovation of broadband technology and standard. Currently, users whoview digital video contents via mobile devices are increasing quickly.More and more video applications for mobile services, e.g., Yuku andSohu, emerge so as to meet these requirements.

This trend leads to difficulties for a content service provider intransmitting media data with a limited bandwidth. Although the bandwidthis always growing stably, the requirement of data communication on thebandwidth grows more quickly. This increase in the requirement isderived from more and more services connected to the Internet and moreand more user's requirements on the media content. It was shown by CiscoVisual Network Index (2015) that global mobile data communication wasincreased by 69% in 2014. In all the mobile data communication, thevideo data accounted for more than 50% in 2012. Based on the above, itbecomes more and more important to increase the multimedia servicecapability of a mobile communication system and provide a user withhigh-quality services.

For the flow media, a network condition and a playback condition of theuser may change frequently, so an adaptive streaming scheme has beenproposed so as to solve the above problem to some extent. A multi-streamvideo encoder server may encode one video image into streams withdifferent code rates and output the streams. For example, afull-high-definition (FHD) video image (with a resolution of 1920*1080)may be encoded into, and outputted as, a HD stream (with a resolution of1280*720) and a D1 stream (with a resolution of 720*576) with differentresolutions (code rates). A client device may request the video encoderserver for a suitable stream in accordance with its network condition,so as to improve the user experience, both for a high-end user and alow-end user.

Such schemes as smooth streaming (Microsoft Corporation), Hyper TextTransport Protocol (HTTP) adaptive streaming (Apple Inc.) and dynamicflash streaming (Adobe Systems Inc.) have achieved great success inmarket.

Currently, for an adaptive bit rate scheme, it is necessary for adisplay device at a user side to adjust an input stream, so as to beadapted to the resolution and display an image at the full resolution. Atypical amplification algorithm (e.g., a Bicubic algorithm or a Lanczosalgorithm (which is used to convert a symmetric matrix into a symmetrictridiagonal matrix through orthogonal similarity transformation, in thename of a twentieth-century Hungarian mathematician Cornelius Lanczos)may introduce a visual error (e.g., sawtooth or annular artifacts). Moreimportantly, the processing is performed by the conventionalamplification algorithm based on the received stream, without taking theoriginal multimedia image data into consideration.

Hence, the conventional method for processing the adaptive media servicemay provide bad user experience.

SUMMARY (1) Technical Problem to be Solved

An object of the present disclosure is to provide a method and a devicefor processing an adaptive media service, an encoder and a decoder, soas to improve the user experience.

(2) Technical Solution

The present disclosure provides in some embodiments a method forprocessing an adaptive media service at an encoder, including: a firstacquisition step of acquiring a first data stream including first imageencoding data obtained by encoding a first image sequence, so as toenable a receiver to acquire the first image sequence in accordance withthe first image encoding data; a second acquisition step of acquiring atleast one second data stream, different second data streams havingdifferent image quality and each second data steam including secondimage encoding data obtained by encoding a second image sequence and atarget optimization parameter corresponding to the second image encodingdata, so as to enable the receiver to decode the second image encodingdata to obtain the second image sequence, and perform qualityimprovement on the second image sequence in accordance with the targetoptimization parameter to obtain a third image sequence, the targetoptimization parameter being obtained in accordance with the first imagesequence and the second image sequence, an identical content beingrecorded in the first image sequence, the second image sequence and thethird image sequence, image quality of each second image sequence beinglower than those of the first image sequence and the third imagesequence; a first selection step of selecting one data stream from afirst data stream set in accordance with a condition of the receiver,the first data stream set at least including the first data stream andthe at least one second data stream; and a first transmission step oftransmitting the selected data stream to the receiver.

In one possible embodiment, the target optimization parameter is anoptimization parameter selected from at least two available optimizationparameters in_param so as to provide MLM(LR,in_param) and the firstimage sequence with a maximum similarity, where LR represents the secondimage sequence, and MLM(LR,in_param) represents an image sequenceobtained through the quality improvement on the LR in accordance withthe available optimization parameter in_param.

In one possible embodiment, the second acquisition step includes: aquality reduction step of performing quality reduction on the firstimage sequence so as to obtain the at least one second image sequence;an encoding step of encoding each of second image sequences so as toobtain the second image encoding data corresponding to each of thesecond image sequences in a one-to-one manner; a first parameterdetermination step of calculating the target optimization parametercorresponding to each of the second image sequences in a one-to-onemanner; and a combination step of combining the second image encodingdata and the corresponding target optimization parameter so as to obtainthe at least one second data stream.

In one possible embodiment, the second acquisition step includes: aquality reduction step of performing quality reduction on the firstimage sequence so as to obtain the at least one second image sequence;an encoding step of encoding each of second image sequences so as toobtain the second image encoding data corresponding to each of thesecond image sequences in a one-to-one manner; a second parameterdetermination step of determining an image type of the first imagesequence and a quality-reduction level corresponding to each of thesecond image sequences in a one-to-one manner; a third parameterdetermination step of determining the target optimization parametercorresponding to each of the second image sequences in accordance with apre-stored correspondence among the quality-reduction level, the imagetype and the target optimization parameter; and a combining step ofcombining each of the second image encoding data and the correspondingtarget optimization parameter so as to obtain the at least one seconddata stream.

In one possible embodiment, the second acquisition step includes: aquality reduction step of performing quality reduction on the firstimage sequence so as to obtain the at least one second image sequence;an encoding step of encoding each of second image sequences so as toobtain the second image encoding data corresponding to each of thesecond image sequences in a one-to-one manner; a first determinationstep of determining a service type of the adaptive media service; afourth parameter determination step of, in the case that the servicetype of the adaptive media service is a real-time service, acquiring thetarget optimization parameter corresponding to each of the second imagesequences in accordance with a pre-stored correspondence among aquality-reduction level, an image type and the target optimizationparameter, and otherwise, calculating the target optimization parametercorresponding to each of the second image sequences; and a combiningstep of combining each of the second image encoding data and thecorresponding target optimization parameter, so as to obtain the atleast one second data stream.

In one possible embodiment, the method further includes a compressionstep of compressing the target optimization parameter corresponding toeach of the second image sequences, and the combining step includescombining the second image encoding data and the correspondingcompressed target optimization parameter.

In one possible embodiment, the second data stream includes a meta datasection and an attachment support section, and the target optimizationparameter is stored in the meta data section or the attachment supportsection.

In one possible embodiment, the method further includes: a thirdacquisition step of acquiring at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a first switching step of,in the case that a data volume of the target optimization parameter isgreater than a predetermined threshold, entering the first selectionstep and otherwise entering the second selection step. The secondselection step includes selecting one data stream from a second datastream set, and the second data stream set consists of the first datastream and the at least one second data stream. The first data streamset consists of the first data stream, the at least one second datastream and the at least one third data stream.

In one possible embodiment, the method further includes a thirdacquisition step of acquiring at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter. The first data stream setfurther includes the at least one third data stream.

In one possible embodiment, the method further includes: a thirdacquisition step of acquiring at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a second switching step of,in the case that the receiver is capable of parsing the targetoptimization parameter and performing the quality improvement inaccordance with the target optimization parameter, entering the firstselection step, and otherwise entering a third selection step ofselecting one data stream from a third data stream set consisting of thefirst data stream and the at least one third data stream. The first datastream set consists of the first data stream and the at least one seconddata stream.

In one possible embodiment, the method further includes: a seconddetermination step of determining, prior to acquiring the at least onesecond data stream, whether or not the target optimization parameterneeds to be updated, so as to acquire a determination result, enteringthe second acquisition step in the case that the determination resultindicates that the target optimization parameter needs to be updated,and otherwise entering a replacement step. The replacement step includesacquiring at least one third data stream corresponding to the at leastone second data stream, selecting one data stream from a third datastream set, and entering the first transmission step. Each third datastream includes the corresponding second image encoding data rather thanthe target optimization parameter. The third data stream set consists ofthe first data stream and the at least one third data stream.

In another aspect, the present disclosure provides in some embodiments amethod for processing an adaptive media service at a decoder, including:a reception step of receiving a second data stream selected by atransmitter in accordance with a condition of a receiver, the seconddata stream including a first section for transmitting second imageencoding data and a second section for transmitting a targetoptimization parameter; a parsing step of parsing the second data streamso as to acquire the second image encoding data carried in the firstsection and the target optimization parameter carried in the secondsection; a decoding step of decoding the second image encoding data soas to obtain a second image sequence, image quality of which is lowerthan that of an original first image sequence; and a quality improvementstep of performing quality improvement on the second image sequence inaccordance with the target optimization parameter, so as to obtain athird image sequence, image quality of which is higher than that of thesecond image sequence.

In one possible embodiment, the method further includes a secondtransmission step of transmitting the condition of the receiver to thetransmitter.

In one possible embodiment, the target optimization parameter is anoptimization parameter selected from at least two available optimizationparameters in_param so as to provide MLM(LR,in_param) and the firstimage sequence with a maximum similarity, where LR represents the secondimage sequence, and MLM(LR,in_param) represents an image sequenceobtained through the quality improvement on the LR in accordance withthe available optimization parameter in_param.

In one possible embodiment, the method further includes a thirdtransmission step of transmitting to the transmitter a messageindicating that the receiver is capable of parsing the targetoptimization parameter and performing the quality improvement inaccordance with the target optimization parameter, so as to enable thetransmitter to generate the second data stream and perform adaptiveselection from a set including the second data stream.

In one possible embodiment, the method further includes: a storage stepof storing the parsed target optimization parameter; and an extractionstep of extracting, prior to receiving a new target optimizationparameter, the stored target optimization parameter for the qualityimprovement step.

In one possible embodiment, the target optimization parameter carried inthe second section is a compressed target optimization parameter, andthe target optimization parameter is acquired in the parsing stepthrough decompression.

In yet another aspect, the present disclosure provides in someembodiments a device for processing an adaptive media service at anencoder, including: a first acquisition module configured to acquire afirst data stream including first image encoding data obtained byencoding a first image sequence, so as to enable a receiver to acquirethe first image sequence in accordance with the first image encodingdata; a second acquisition module configured to acquire at least onesecond data stream, different second data streams having different imagequality and each second data steam including second image encoding dataobtained by encoding a second image sequence and a target optimizationparameter corresponding to the second image encoding data, so as toenable the receiver to decode the second image encoding data to obtainthe second image sequence, and perform quality improvement on the secondimage sequence in accordance with the target optimization parameter toobtain a third image sequence, the target optimization parameter beingobtained in accordance with the first image sequence and the secondimage sequence, an identical content being recorded in the first imagesequence, the second image sequence and the third image sequence, imagequality of each second image sequence being lower than those of thefirst image sequence and the third image sequence; a first selectionmodule configured to select one data stream from a first data stream setin accordance with a condition of the receiver, the first data streamset at least including the first data stream and the at least one seconddata stream; and a first transmission module configured to transmit theselected data stream to the receiver.

In one possible embodiment, the target optimization parameter is anoptimization parameter selected from at least two available optimizationparameters in_param so as to provide MLM(LR,in_param) and the firstimage sequence with a maximum similarity, where LR represents the secondimage sequence, and MLM(LR,in_param) represents an image sequenceobtained through the quality improvement on the LR in accordance withthe available optimization parameter in_param.

In one possible embodiment, the second acquisition module includes: aquality reduction module configured to perform quality reduction on thefirst image sequence so as to obtain the at least one second imagesequence; an encoding module configured to encode each of second imagesequences so as to obtain the second image encoding data correspondingto each of the second image sequences in a one-to-one manner; a firstparameter determination module configured to calculate the targetoptimization parameter corresponding to each of the second imagesequences in a one-to-one manner; and a combination module configured tocombine the second image encoding data and the corresponding targetoptimization parameter so as to obtain the at least one second datastream.

In one possible embodiment, the second acquisition module includes: aquality reduction module configured to perform quality reduction on thefirst image sequence so as to obtain the at least one second imagesequence; an encoding module configured to encode each of second imagesequences so as to obtain the second image encoding data correspondingto each of the second image sequences in a one-to-one manner; a secondparameter determination module configured to determine an image type ofthe first image sequence and a quality-reduction level corresponding toeach of the second image sequences in a one-to-one manner; a thirdparameter determination module configured to determine the targetoptimization parameter corresponding to each of the second imagesequences in accordance with a pre-stored correspondence among thequality-reduction level, the image type and the target optimizationparameter; and a combination module configured to combine each of thesecond image encoding data and the corresponding target optimizationparameter so as to obtain the at least one second data stream.

In one possible embodiment, the second acquisition module includes: aquality reduction module configured to perform quality reduction on thefirst image sequence so as to obtain the at least one second imagesequence; an encoding module configured to encode each of second imagesequences so as to obtain the second image encoding data correspondingto each of the second image sequences in a one-to-one manner; a firstdetermination module configured to determine a service type of theadaptive media service; a fourth parameter determination moduleconfigured to, in the case that the service type of the adaptive mediaservice is a real-time service, acquire the target optimizationparameter corresponding to each of the second image sequences inaccordance with a pre-stored correspondence among a quality-reductionlevel, an image type and the target optimization parameter, andotherwise, calculate the target optimization parameter corresponding toeach of the second image sequences; and a combination module configuredto combine each of the second image encoding data and the correspondingtarget optimization parameter, so as to obtain the at least one seconddata stream.

In one possible embodiment, the device further includes a compressionmodule configured to compress the target optimization parametercorresponding to each of the second image sequences, and the combinationmodule is further configured to combine the second image encoding dataand the corresponding compressed target optimization parameter.

In one possible embodiment, the second data stream includes a meta datasection and an attachment support section, and the target optimizationparameter is stored in the meta data section or the attachment supportsection.

In one possible embodiment, the device further includes: a thirdacquisition module configured to acquire at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a first switching moduleconfigured to, in the case that a data volume of the target optimizationparameter is greater than a predetermined threshold, trigger the firstselection module and otherwise trigger the second selection module. Thesecond selection module is further configured to select one data streamfrom a second data stream set, and the second data stream set consistsof the first data stream and the at least one second data stream. Thefirst data stream set consists of the first data stream, the at leastone second data stream and the at least one third data stream.

In one possible embodiment, the device further includes a thirdacquisition module configured to acquire at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter. The first data stream setfurther includes the at least one third data stream.

In one possible embodiment, the device further includes: a thirdacquisition module configured to acquire at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a second switching moduleconfigured to, in the case that the receiver is capable of parsing thetarget optimization parameter and performing the quality improvement inaccordance with the target optimization parameter, trigger the firstselection module, and otherwise trigger a third selection module whichis configured to select one data stream from a third data stream setconsisting of the first data stream and the at least one third datastream. The first data stream set consists of the first data stream andthe at least one second data stream.

In still yet another aspect, the present disclosure provides in someembodiments a device for processing an adaptive media service at adecoder, including: a reception module configured to receive a seconddata stream selected by a transmitter in accordance with a condition ofa receiver, the second data stream including a first section fortransmitting second image encoding data and a second section fortransmitting a target optimization parameter; a parsing moduleconfigured to parse the second data stream so as to acquire the secondimage encoding data carried in the first section and the targetoptimization parameter carried in the second section; a decoding moduleconfigured to decode the second image encoding data so as to obtain asecond image sequence, image quality of which is lower than that of anoriginal first image sequence; and a quality improvement moduleconfigured to perform quality improvement on the second image sequencein accordance with the target optimization parameter, so as to obtain athird image sequence, image quality of which is higher than that of thesecond image sequence.

In one possible embodiment, the device further includes a secondtransmission module configured to transmit the condition of the receiverto the transmitter.

In one possible embodiment, the target optimization parameter is anoptimization parameter selected from at least two available optimizationparameters in_param so as to provide MLM(LR,in_param) and the firstimage sequence with a maximum similarity, where LR represents the secondimage sequence, and MLM(LR,in_param) represents an image sequenceobtained through the quality improvement on the LR in accordance withthe available optimization parameter in_param.

In one possible embodiment, the device further includes a thirdtransmission module configured to transmit to the transmitter a messageindicating that the receiver is capable of parsing the targetoptimization parameter and performing the quality improvement inaccordance with the target optimization parameter, so as to enable thetransmitter to generate the second data stream and perform adaptiveselection from a set including the second data stream.

In one possible embodiment, the device further includes: a storagemodule configured to store the parsed target optimization parameter; andan extraction module configured to extract, prior to receiving a newtarget optimization parameter, the stored target optimization parameterfor the quality improvement module.

In one possible embodiment, the target optimization parameter carried inthe second section is a compressed target optimization parameter, andthe target optimization parameter is acquired by the parsing modulethrough decompression.

In still yet another aspect, the present disclosure provides in someembodiments an encoder including the above-mentioned device forprocessing an adaptive media service at an encoder.

In still yet another aspect, the present disclosure provides in someembodiments a decoder including the above-mentioned device forprocessing an adaptive media service at a decoder.

(3) Beneficial Effects

At least one of the following beneficial effects may be acquired.

Firstly, according to the embodiments of the present disclosure, thetarget optimization parameter obtained in accordance with an originalimage sequence and an image sequence carried in a current stream may beadded into the stream, so that the quality improvement may be performedon an image sequence decoded from the stream at the decoder inaccordance with the target optimization parameter, so as to acquire animage sequence having higher image quality. Because the original imagesequence is taken into consideration in the target optimizationparameter, as compared with the image sequence obtained merely throughoptimizing the image sequence decoded from the stream, it is able toprovide higher image quality and thereby improve the user experience.

Secondly, according to the embodiments of the present disclosure, theimage sequence decoded from the stream may be optimized based on thesimilarity to the original image sequence, and the target optimizationparameter may be selected so as to obtain the image sequence having thehighest similarity to the original image sequence. As a result, it isable to further improve the user experience.

Thirdly, according to the embodiments of the present disclosure, thetarget optimization parameter may be carried in the meta data section orthe attachment support section, so a client device that does not supportthe methods in the embodiments of the present disclosure may neglect thetarget optimization parameter and perform the processing using theconventional method. As a result, it is able to achieve forwardcompatibility.

Fourthly, according to the embodiments of the present disclosure, thetarget optimization parameter may be compressed and combined into thestream for the subsequent transmission, so it is able to reduce therequirement on a network bandwidth.

Fifthly, according to the embodiments of the present disclosure,different streams may be generated depending on whether or not theclient device supports the methods in the embodiments of the presentdisclosure. As a result, it is able to not only reduce the burden at aserver, but also improve a switching efficiency and a switching speed.

Sixthly, according to the embodiments of the present disclosure, thequality improvement may be performed in accordance with thepreviously-received target optimization parameter in the case that anapplication scenario is not changed. As a result, it is able to reducethe data volume transmitted between a server and the client.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to illustrate the technical solutions in the embodiments of thepresent disclosure or the related art in a clearer manner, the drawingsdesired for the present disclosure or the related art will be describedbriefly hereinafter. Obviously, the following drawings merely relate toparts of the embodiments of the present disclosure, and a person skilledin the art may, without any creative effort, acquire the other drawingsbased on these drawings.

FIG. 1 is a flow chart of a method for processing an adaptive mediaservice at an encoder according to one embodiment of the presentdisclosure;

FIG. 2 is a flow chart of a second acquisition step according to oneembodiment of the present disclosure;

FIG. 3 is another flow chart of the second acquisition step according toone embodiment of the present disclosure;

FIG. 4 is yet another flow chart of the second acquisition stepaccording to one embodiment of the present disclosure;

FIG. 5 is another flow chart of the method for processing an adaptivemedia service at an encoder according to one embodiment of the presentdisclosure;

FIG. 6 is yet another flow chart of the method for processing anadaptive media service at an encoder according to one embodiment of thepresent disclosure;

FIG. 7 is still yet another flow chart of the method for processing anadaptive media service at an encoder according to one embodiment of thepresent disclosure;

FIG. 8 is a flow chart of a method for processing an adaptive mediaservice at a decoder according to one embodiment of the presentdisclosure;

FIG. 9 is a schematic view showing a device for processing an adaptivemedia service at an encoder according to one embodiment of the presentdisclosure;

FIG. 10 is another schematic view showing the device for processing anadaptive media service at an encoder according to one embodiment of thepresent disclosure;

FIG. 11 is a schematic view showing a device for processing an adaptivemedia service at a decoder according to one embodiment of the presentdisclosure;

FIG. 12 is a schematic view showing a service system for an off-lineservice at an encoder according to one embodiment of the presentdisclosure;

FIG. 13 is a schematic view showing the service system for a real-timeservice at an encoder according to one embodiment of the presentdisclosure; and

FIG. 14 is a schematic view showing a service system at a decoderaccording to one embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure will be described hereinafter in conjunction withthe drawings and embodiments. The following embodiments are forillustrative purposes only, but shall not be used to limit the scope ofthe present disclosure.

In order to make the objects, the technical solutions and the advantagesof the present disclosure more apparent, the present disclosure will bedescribed hereinafter in a clear and complete manner in conjunction withthe drawings and embodiments. Obviously, the following embodimentsmerely relate to a part of, rather than all of, the embodiments of thepresent disclosure, and based on these embodiments, a person skilled inthe art may, without any creative effort, obtain the other embodiments,which also fall within the scope of the present disclosure.

Unless otherwise defined, any technical or scientific term used hereinshall have the common meaning understood by a person of ordinary skills.Such words as “first” and “second” used in the specification and claimsare merely used to differentiate different components rather than torepresent any order, number or importance. Similarly, such words as“one” or “one of” are merely used to represent the existence of at leastone member, rather than to limit the number thereof. Such words as“connect” or “connected to” may include electrical connection, direct orindirect, rather than to be limited to physical or mechanicalconnection. Such words as “on”, “under”, “left” and “right” are merelyused to represent relative position relationship, and when an absoluteposition of the object is changed, the relative position relationshipwill be changed too.

According to the embodiments of the present disclosure, a targetoptimization parameter obtained in accordance with an original imagesequence and an image sequence carried in a current stream may be addedinto the stream, so that the quality improvement may be performed on animage sequence decoded from the stream at a decoder in accordance withthe target optimization parameter, so as to acquire an image sequencehaving higher image quality. Because the original image sequence istaken into consideration in the target optimization parameter, ascompared with the image sequence obtained merely through optimizing theimage sequence decoded from the stream, it is able to provide higherimage quality and thereby improve the user experience.

As shown in FIG. 1, the present disclosure provides in some embodimentsa method for processing an adaptive media service, which includes: afirst acquisition step 101 of acquiring a first data stream includingfirst image encoding data obtained by encoding a first image sequence,so as to enable a receiver to acquire the first image sequence inaccordance with the first image encoding data; a second acquisition step102 of acquiring at least one second data stream, different second datastreams having different image quality and each second data steamincluding second image encoding data obtained by encoding a second imagesequence and a target optimization parameter corresponding to the secondimage encoding data, so as to enable the receiver to decode the secondimage encoding data to obtain the second image sequence, and performquality improvement on the second image sequence in accordance with thetarget optimization parameter to obtain a third image sequence, thetarget optimization parameter being obtained in accordance with thefirst image sequence and the second image sequence, an identical contentbeing recorded in the first image sequence, the second image sequenceand the third image sequence, image quality of each second imagesequence being lower than those of the first image sequence and thethird image sequence; a first selection step 103 of selecting one datastream from a first data stream set in accordance with a condition ofthe receiver, the first data stream set at least including the firstdata stream and the at least one second data stream; and a firsttransmission step 104 of transmitting the selected data stream to thereceiver.

According to the embodiments of the present disclosure, the targetoptimization parameter obtained in accordance with an original imagesequence and an image sequence carried in a current stream may be addedinto the stream, so that the quality improvement may be performed on animage sequence decoded from the stream at the decoder in accordance withthe target optimization parameter, so as to acquire an image sequencehaving higher image quality. Because the original image sequence istaken into consideration in the target optimization parameter, ascompared with the image sequence obtained merely through optimizing theimage sequence decoded from the stream, it is able to provide higherimage quality and thereby improve the user experience.

In the embodiments of the present disclosure, the various optimizationmodes may be used, and one possible optimization mode will be describedhereinafter.

In an alternative embodiment of the present disclosure, the targetoptimization parameter is an optimization parameter selected from atleast two available optimization parameters in_param so as to provideMLM(LR,in_param) and the first image sequence with a maximum similarity,where LR represents the second image sequence, and MLM(LR,in_param)represents an image sequence obtained through the quality improvement onthe LR in accordance with the available optimization parameter in_param.

In other words, the target optimization parameter may be determined inaccordance with such a condition that a first similarity is greater thanany second similarity. Of course, the target optimization parameter mayalso be determined in accordance with any other condition similar to thesimilarity, which will not be particularly defined herein.

The first similarity is that between the third image sequence and thefirst image sequence, and the second similarity is that between an imagesequence obtained through quality improvement on the second imagesequence in accordance with the other optimization parameter in theavailable optimization parameters different from the target optimizationparameter and the first image sequence.

The target optimization parameter will be described hereinaftermathematically.

In the embodiments of the present disclosure, for a k^(th) stream, thetarget optimization parameter out_param may be represented as follows:out_param≈ArgMax_(param)Similarity(MLM(LR[k],in_param[k]),HR[k]), whereArgMaxx(f(x)) represents a variable x corresponding to a maximum valueof f(x), LR[k] represents an image sequence carried in the k^(th)stream, i.e., the corresponding second image sequence in the k^(th)stream, In_param[k] represents available optimization parameterscorresponding to the k^(th) stream and there are at least two availableoptimization parameters, HR[k] represents an input image sequencecorresponding to the k^(th) stream, i.e., the original first imagesequence to be transmitted, and MLM(LR[k],in_param[k]) represents theimage sequence obtained through the quality improvement on LR[k] inaccordance with the available optimization parameter In_param[k].

For example, an algorithm set may include algorithms Z1, Z2, Z3, . . . ,and Zn (i.e., the optimization parameters are the algorithms used forperforming the quality improvement on the image). For an image sequenceY carried in a to-be-transmitted stream (the image sequence Y isobtained through quality reduction on an original image sequence X), thequality improvement may be performed (in real time or in advance, whichwill be described hereinafter) on its image sequence Y in accordancewith Z1, Z2, Z3, . . . , and Zn, so as to obtain image sequences Y1, Y2,Y3, . . . , and Yn.

At this time, similarities between Y1, Y2, Y3, . . . , Yn and X may becalculated respectively, so as to determine an image sequence Ym (m=1,2, . . . , n) with the highest similarity.

At this time, an algorithm corresponding to the image sequence Ym is anoptimum algorithm, and the target optimization parameter may bedetermined as the algorithm corresponding to the image sequence Ym. At adecoder, the corresponding quality improvement is performed on the imagesequence carried in the kth stream using the algorithm represented bythe target optimization parameter.

In the embodiments of the present disclosure, these algorithms may belinear interpolation algorithms, bicubic interpolation algorithms orcellular neural network (CNN) algorithms, which will not be particularlydefined herein.

For the CNN algorithm, a parameter for 3 CNN layers needs to beprovided, so as to enable a CNN to perform the quality improvement onthe image.

The training of the CNN and the determination of the parameter arebeyond the scope of the present disclosure, and thus will not beparticularly defined herein.

The adaptive media service may include two types of services, i.e., anoff-line service and a real-time service (e.g., a video conference).Among them, the real-time performance for the transmission of the imagesequence is highly demanded by the real-time service. Hence, in theembodiments of the present disclosure, with respect to the two types ofservices with different real-time requirements, the corresponding targetoptimization parameters may be acquired in different modes. Of course,in the case of lower real-time requirement, the corresponding targetoptimization parameter may be acquired in an identical mode, which willnot be particularly defined herein.

It is found that, in the embodiments of the present disclosure, a seriesof operations, including an operation for the quality improvement and anoperation for the similarity of the image sequence, need to be performedso as to determine the target optimization parameters corresponding todifferent streams. As is known to all, it may take a certain period oftime for any operation, and especially in the case of an inferiorprocessor, it may take more time to perform the operations.

For the off-line service, its data content (e.g., a movie) alreadyexists, so an encoding operation may be performed prior to a requestfrom a user. At this time, the data content may also be played by theclient before the entire file has been completely downloaded. Hence, forthe off-line service with lower real-time requirement, the targetoptimization parameter may be calculated in real time during thetransmission.

In this regard, the second acquisition step 102 may include: a qualityreduction step 1021 of performing quality reduction on the first imagesequence so as to obtain the at least one second image sequence; anencoding step 1022 of encoding each of second image sequences so as toobtain the second image encoding data corresponding to each of thesecond image sequences in a one-to-one manner; a first parameterdetermination step 1023 of calculating the target optimization parametercorresponding to each of the second image sequences in a one-to-onemanner; and a combination step 1024 of combining the second imageencoding data and the corresponding target optimization parameter so asto obtain the at least one second data stream.

As mentioned above, the second acquisition step includes the firstparameter determination step, so it is necessary to calculate in realtime the target optimization parameter corresponding to each of thesecond image sequences, i.e., perform the quality improvement on thesecond image sequences in accordance with each algorithm in thealgorithm set, then calculate the similarity between each of imagesequences obtained after the quality improvement and the first imagesequence, and then select the corresponding target optimizationparameter in accordance with the similarity.

It is found that, although with a large computational burden, it is ableto acquire the target optimization parameter in a more accurate manner.

For the real-time service with high real-time requirement, theabove-mentioned mode for acquiring the target optimization parameter maybe not particularly suitable. In this regard, the present disclosureprovides in some embodiments another mode for acquiring the targetoptimization parameter through calculating the target optimizationparameter in advance and then looking up a table, so as to meet thereal-time requirement.

As shown in FIG. 3, the second acquisition step 102 includes: a qualityreduction step 1021 of performing quality reduction on the first imagesequence so as to obtain the at least one second image sequence; anencoding step 1022 of encoding each of second image sequences so as toobtain the second image encoding data corresponding to each of thesecond image sequences in a one-to-one manner; a second parameterdetermination step 1025 of determining an image type of the first imagesequence and a quality-reduction level corresponding to each of thesecond image sequences in a one-to-one manner; a third parameterdetermination step 1026 of determining the target optimization parametercorresponding to each of the second image sequences in accordance with apre-stored correspondence among the quality-reduction level, the imagetype and the target optimization parameter; and a combining step 1024 ofcombining each of the second image encoding data and the correspondingtarget optimization parameter so as to obtain the at least one seconddata stream.

It should be appreciated that, the serial numbers of the above steps arenot used to represent any order of the steps.

It is found that, in the above mode for acquiring the targetoptimization parameter, prior to acquiring the second data stream, it ismerely required to determine the image type of the first image sequenceand the quality-reduction level corresponding to each of the secondimage sequences, and then determine the target optimization parameter inaccordance with the correspondence among the pre-storedquality-reduction level, the image type and the target optimizationparameter, so as to remarkably reduce the computational burden ascompared with the mode where the quality improvement and the calculationof the similarity between the image sequences need to be performed. As aresult, it is able to improve a processing speed and meet the real-timerequirement of the real-time service.

For example, there are N1 image types (e.g., cartoon or naturallandscape), and the original image sequence has N2 resolutions. Inaddition, with respect to each resolution of the original imagesequence, the second image sequence has N3 resolutions (which are alllower than the resolution of the original image sequence), i.e., N3 isless than N2.

Here, the quality-reduction level refers to the conversion between theresolution of the original image sequence and the resolution of thesecond image sequence.

In the case that the original image sequence has a resolution of1920*1080 and the second image sequence has resolutions of 1280*720 and720*576, there may be two quality-reduction levels, i.e., a level from1920*1080 to 1280*720, and a level from 1920*1080 to 720*576.

In this case, in the above-mentioned second mode for acquiring thetarget optimization parameter, it is required to calculate in advancethe corresponding target optimization parameters with respect toN1*N2*N3 combinations, so as to obtain the correspondence among thequality-reduction levels, the image types and the target optimizationparameters.

The acquired correspondence may be stored in a database. After acquiringthe image type of the first image sequence and the quality-reductionlevel corresponding to the second image sequence, it is able todetermine the target optimization parameter in accordance with thecorrespondence.

The above-mentioned two modes for acquiring the target optimizationparameter may be used independently, or in combination so as to beadapted to various service types. In other words, as shown in FIG. 4,the second acquisition step may include: a quality reduction step 1021of performing quality reduction on the first image sequence so as toobtain the at least one second image sequence; an encoding step 1022 ofencoding each of second image sequences so as to obtain the second imageencoding data corresponding to each of the second image sequences in aone-to-one manner; a determination step 1027 of determining a servicetype of the adaptive media service; a fourth parameter determinationstep 1028 of, in the case that the service type of the adaptive mediaservice is a real-time service, acquiring the target optimizationparameter corresponding to each of the second image sequences inaccordance with a pre-stored correspondence among a quality-reductionlevel, an image type and the target optimization parameter, andotherwise, calculating the target optimization parameter correspondingto each of the second image sequences; and a combining step 1024 ofcombining each of the second image encoding data and the correspondingtarget optimization parameter, so as to obtain the at least one seconddata stream.

In this mode, the target optimization parameter may be acquiredadaptively at the encoder in accordance with the service type. In thecase that the service type is an off-line service, the targetoptimization parameter may be acquired through calculation in real time,so as to improve the quality of the image sequence outputted at theencoder. In the case that the service type is a real-time service, thetarget optimization parameter may be acquired directly in accordancewith the pre-stored correspondence, so as to meet the real-timerequirement.

As compared with the related art, in the embodiments of the presentdisclosure, additional data (the target optimization parameter) needs tobe transmitted. Hence, the target optimization parameter may becompressed, so as to reduce the data transmission volume.

In other words, the method may further include a compression step ofcompressing the target optimization parameter corresponding to each ofthe second image sequences, and the combining step includes combiningthe second image encoding data and the corresponding compressed targetoptimization parameter.

In the embodiments of the present disclosure, the additional data (thetarget optimization parameter) may be transmitted in various ways, e.g.,adding a field in the data stream or using a reserved field. However, itis impossible for these ways to achieve forward compatibility.

In the embodiments of the present disclosure, in order to achieve theforward compatibility, the target optimization parameter may be carriedin a meta data section or an attachment support section having theforward compatibility. In this way, a client that does not support themethod in the embodiments of the present disclosure may neglect thetarget optimization parameter and perform the processing through theconventional method.

In other words, in the embodiments of the present disclosure, the seconddata stream including the meta data section and the attachment supportsection, and the target optimization parameter is stored in the metadata section or the attachment support section.

No influence may be caused to the receiver in the case that the targetoptimization parameter is transmitted via the meta data section or theattachment support section. Hence, in the case that the targetoptimization parameter is of a small data volume, no great influence maybe caused even in the case that the target optimization parameter iscarried in the data stream. At this time, the target optimizationparameters may be directly carried in all the data streams, so as toreduce the number of the streams, and improve the efficiency at theencoder and the switching efficiency between the streams.

In the case that the target optimization parameter is of a great datavolume, a great influence may be caused in the case that the targetoptimization parameter is carried in the data stream. At this time, theswitching efficiency is no longer the most important consideration, andit is required to provide each stream with the data stream carrying thetarget optimization parameter and the data stream not carrying thetarget optimization parameter.

As mentioned above, in the above-mentioned way, both the switchingefficiency and the transmission efficiency are taken into consideration.

In other words, as shown in FIG. 5, the method may further include: athird acquisition step 105 of acquiring at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a first switching step 106of, in the case that a data volume of the target optimization parameteris greater than a predetermined threshold, entering the first selectionstep 103 and otherwise entering the second selection step 107. Thesecond selection step 107 includes selecting one data stream from asecond data stream set, and the second data stream set consists of thefirst data stream and the at least one second data stream. The firstdata stream set consists of the first data stream, the at least onesecond data stream and the at least one third data stream.

Of course, the stream carrying the target optimization parameter maycertainly have a data volume different from the conventional stream notcarrying the target optimization parameter, and thereby the selection ofthe stream may be adversely affected. Hence, in the embodiments of thepresent disclosure, in another way, a new stream carrying the targetoptimization parameter and the conventional stream not carrying thetarget optimization parameter may be generated with respect to eachimage sequence, so as to improve the flexibility of the adaptiveselection.

In other words, the method may further include a third acquisition stepof acquiring at least one third data stream corresponding to the atleast one second data stream, each third data stream including thecorresponding second image encoding data rather than the targetoptimization parameter. The first data stream set further includes theat least one third data stream.

As mentioned above, the target optimization parameter may be transmittedin various ways, but in the case that an opposite end (the receiver)does not support the quality improvement in accordance with the targetoptimization parameter, additional invalid data may occur even if noinfluence is caused by the target optimization parameter to thereceiver.

Hence, in the embodiments of the present disclosure, whether or not thetarget optimization parameter is to be transmitted may depend on whetheror not the receiver supports the target optimization parameter, so as toimprove the utilization rate of the bandwidth.

In other words, as shown in FIG. 6, the method may further include: athird acquisition step 105 of acquiring at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a second switching step 108of, in the case that the receiver is capable of parsing the targetoptimization parameter and performing the quality improvement inaccordance with the target optimization parameter, entering the firstselection step 103, and otherwise entering a third selection step 109 ofselecting one data stream from a third data stream set consisting of thefirst data stream and the at least one third data stream. The first datastream set consists of the first data stream and the at least one seconddata stream.

In the embodiments of the present disclosure, a network condition andany other playing condition of the receiver may be changed gradually,and the streams shall not be switched frequently. Hence, the targetoptimization parameter may also be transmitted periodically or merely inthe case that it has been changed, so as to reduce the data transmissionvolume. At this time, it is required to store a historical targetoptimization parameter at the receiver, and update it in real time. Inthe case of no new target optimization parameter, the stored historicaltarget optimization parameter may be used for the subsequent qualityimprovement.

At this time, as shown in FIG. 7, the method may further include asecond determination step 110 of determining, prior to acquiring the atleast one second data stream, whether or not the target optimizationparameter needs to be updated, so as to acquire a determination result,entering the second acquisition step 102 in the case that thedetermination result indicates that the target optimization parameterneeds to be updated, and otherwise entering a replacement step 111. Thereplacement step 111 includes acquiring at least one third data streamcorresponding to the at least one second data stream, selecting one datastream from a third data stream set, and entering the first transmissionstep. Each third data stream includes the corresponding second imageencoding data rather than the target optimization parameter. The thirddata stream set consists of the first data stream and the at least onethird data stream.

In this way, it is able to further reduce the data transmission volumecaused by the target optimization parameter.

Whether or not the target optimization parameter is to be updated maydepend on various conditions, e.g., when an update period is expired, orwhen a transmission scenario has been changed, or when a stream selectedby the receiver has been changed.

The above description is given by taking the encoder as an example. Themethod for processing the adaptive media service will be furtherdescribed hereinafter from the aspect of a decoder.

As shown in FIG. 8, the present disclosure further provides in someembodiments a method for processing an adaptive media service at adecoder, which may include: a reception step 201 of receiving a seconddata stream selected by a transmitter in accordance with a condition ofa receiver, the second data stream including a first section fortransmitting second image encoding data and a second section fortransmitting a target optimization parameter; a parsing step 202 ofparsing the second data stream so as to acquire the second imageencoding data carried in the first section and the target optimizationparameter carried in the second section; a decoding step 203 of decodingthe second image encoding data so as to obtain a second image sequence,image quality of which is lower than that of an original first imagesequence; and a quality improvement step 204 of performing qualityimprovement on the second image sequence in accordance with the targetoptimization parameter, so as to obtain a third image sequence, imagequality of which is higher than that of the second image sequence.

The target optimization parameter may be transmitted by the transmitteronly in the case that the receiver is capable of parsing the targetoptimization parameter and performing the quality improvement inaccordance with the target optimization parameter, so as to improve theutilization rate of the network bandwidth. At this time, the method mayfurther include a second transmission step of transmitting the conditionof the receiver to the transmitter.

In the embodiments of the present disclosure, the target optimizationparameter may be determined as follows. The target optimizationparameter is an optimization parameter selected from at least twoavailable optimization parameters in_param so as to provideMLM(LR,in_param) and the first image sequence with a maximum similarity,where LR represents the second image sequence, and MLM(LR,in_param)represents an image sequence obtained through the quality improvement onthe LR in accordance with the available optimization parameter in_param.

In other words, the target optimization parameter may be determined inaccordance with such a condition that a first similarity is greater thanany second similarity. The first similarity is that between the thirdimage sequence and the first image sequence, and the second similarityis that between an image sequence obtained through quality improvementon the second image sequence in accordance with the other optimizationparameter in the available optimization parameters different from thetarget optimization parameter and the first image sequence.

The determination of the target optimization parameter has beendescribed hereinabove, and thus will not be particularly defined herein.

In order to enable the transmitter to transmit the target optimizationparameter only in the case that the receiver is capable of parsing thetarget optimization parameter and performing the quality improvement inaccordance with the target optimization parameter, thereby to improvethe utilization rate of the network bandwidth, the method may furtherinclude a third transmission step of transmitting to the transmitter amessage indicating that the receiver is capable of parsing the targetoptimization parameter and performing the quality improvement inaccordance with the target optimization parameter, so as to enable thetransmitter to generate the second data stream and perform adaptiveselection from a set including the second data stream.

In the embodiments of the present disclosure, a network condition andany other playing condition of the receiver may be changed gradually,and the streams shall not be switched frequently. Hence, the targetoptimization parameter may also be transmitted periodically or merely inthe case that it has been changed, so as to reduce the data transmissionvolume. At this time, it is required to store a historical targetoptimization parameter at the receiver, and update it in real time. Inthe case of no new target optimization parameter, the stored historicaltarget optimization parameter may be used for the subsequent qualityimprovement.

In other words, the method may further include: a storage step ofstoring the parsed target optimization parameter; and an extraction stepof extracting, prior to receiving a new target optimization parameter,the stored target optimization parameter for the quality improvementstep.

Of course, in order to reduce the data transmission volume caused by thetarget optimization parameter, in the embodiments of the presentdisclosure, the target optimization parameter carried in the secondsection is a compressed target optimization parameter, and the targetoptimization parameter is acquired in the parsing step throughdecompression.

As shown in FIG. 9, the present disclosure further provides in someembodiments a device for processing an adaptive media service at anencoder, which may include: a first acquisition module configured toacquire a first data stream including first image encoding data obtainedby encoding a first image sequence, so as to enable a receiver toacquire the first image sequence in accordance with the first imageencoding data; a second acquisition module configured to acquire atleast one second data stream, different second data streams havingdifferent image quality and each second data steam including secondimage encoding data obtained by encoding a second image sequence and atarget optimization parameter corresponding to the second image encodingdata, so as to enable the receiver to decode the second image encodingdata to obtain the second image sequence, and perform qualityimprovement on the second image sequence in accordance with the targetoptimization parameter to obtain a third image sequence, the targetoptimization parameter being obtained in accordance with the first imagesequence and the second image sequence, an identical content beingrecorded in the first image sequence, the second image sequence and thethird image sequence, image quality of each second image sequence beinglower than those of the first image sequence and the third imagesequence; a first selection module configured to select one data streamfrom a first data stream set in accordance with a condition of thereceiver, the first data stream set at least including the first datastream and the at least one second data stream; and a first transmissionmodule configured to transmit the selected data stream to the receiver.

In one possible embodiment, the target optimization parameter is anoptimization parameter selected from at least two available optimizationparameters in_param so as to provide MLM(LR,in_param) and the firstimage sequence with a maximum similarity, where LR represents the secondimage sequence, and MLM(LR,in_param) represents an image sequenceobtained through the quality improvement on the LR in accordance withthe available optimization parameter in_param.

In other words, the target optimization parameter may be determined inaccordance with such a condition that a first similarity is greater thanany second similarity. The first similarity is that between the thirdimage sequence and the first image sequence, and the second similarityis that between an image sequence obtained through quality improvementon the second image sequence in accordance with the other optimizationparameter in the available optimization parameters different from thetarget optimization parameter and the first image sequence.

In one possible embodiment, the second acquisition module includes: aquality reduction module configured to perform quality reduction on thefirst image sequence so as to obtain the at least one second imagesequence; an encoding module configured to encode each of second imagesequences so as to obtain the second image encoding data correspondingto each of the second image sequences in a one-to-one manner; a firstparameter determination module configured to calculate the targetoptimization parameter corresponding to each of the second imagesequences in a one-to-one manner; and a combination module configured tocombine the second image encoding data and the corresponding targetoptimization parameter so as to obtain the at least one second datastream.

In one possible embodiment, the second acquisition module includes: aquality reduction module configured to perform quality reduction on thefirst image sequence so as to obtain the at least one second imagesequence; an encoding module configured to encode each of second imagesequences so as to obtain the second image encoding data correspondingto each of the second image sequences in a one-to-one manner; a secondparameter determination module configured to determine an image type ofthe first image sequence and a quality-reduction level corresponding toeach second image sequence; a third parameter determination moduleconfigured to determine the target optimization parameter correspondingto each of the second image sequences in accordance with a pre-storedcorrespondence among the quality-reduction level, the image type and thetarget optimization parameter; and a combination module configured tocombine each of the second image encoding data and the correspondingtarget optimization parameter so as to obtain the at least one seconddata stream.

In one possible embodiment, the second acquisition module includes: aquality reduction module configured to perform quality reduction on thefirst image sequence so as to obtain the at least one second imagesequence; an encoding module configured to encode each of second imagesequences so as to obtain the second image encoding data correspondingto each of the second image sequences in a one-to-one manner; a firstdetermination module configured to determine a service type of theadaptive media service; a fourth parameter determination moduleconfigured to, in the case that the service type of the adaptive mediaservice is a real-time service, acquire the target optimizationparameter corresponding to each of the second image sequences inaccordance with a pre-stored correspondence among a quality-reductionlevel, an image type and the target optimization parameter, andotherwise, calculate the target optimization parameter corresponding toeach of the second image sequences; and a combination module configuredto combine each of the second image encoding data and the correspondingtarget optimization parameter, so as to obtain the at least one seconddata stream.

In one possible embodiment, the device further includes a compressionmodule configured to compress the target optimization parametercorresponding to each of the second image sequences, and the combinationmodule is further configured to combine the second image encoding dataand the corresponding compressed target optimization parameter.

In one possible embodiment, the second data stream includes a meta datasection and an attachment support section, and the target optimizationparameter is stored in the meta data section or the attachment supportsection.

In one possible embodiment, the device further includes: a thirdacquisition module configured to acquire at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a first switching moduleconfigured to, in the case that a data volume of the target optimizationparameter is greater than a predetermined threshold, trigger the firstselection module and otherwise trigger the second selection module. Thesecond selection module is further configured to select one data streamfrom a second data stream set, and the second data stream set consistsof the first data stream and the at least one second data stream. Thefirst data stream set consists of the first data stream, the at leastone second data stream and the at least one third data stream.

In one possible embodiment, the device further includes a thirdacquisition module configured to acquire at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter. The first data stream setfurther includes the at least one third data stream.

In one possible embodiment, the device further includes: a thirdacquisition module configured to acquire at least one third data streamcorresponding to the at least one second data stream, each third datastream including the corresponding second image encoding data ratherthan the target optimization parameter; and a second switching moduleconfigured to, in the case that the receiver is capable of parsing thetarget optimization parameter and performing the quality improvement inaccordance with the target optimization parameter, trigger the firstselection module, and otherwise trigger a third selection module whichis configured to select one data stream from a third data stream setconsisting of the first data stream and the at least one third datastream. The first data stream set consists of the first data stream andthe at least one second data stream.

As shown in FIG. 10, the device further includes: a second determinationmodule configured to determine, prior to acquiring the at least onesecond data stream, whether or not the target optimization parameterneeds to be updated, so as to acquire a determination result, triggerthe second acquisition module in the case that the determination resultindicates that the target optimization parameter needs to be updated,and otherwise trigger a replacement module. The replacement module isconfigured to acquire at least one third data stream corresponding tothe at least one second data stream, select one data stream from a thirddata stream set, and trigger the first transmission module. Each thirddata stream includes the corresponding second image encoding data ratherthan the target optimization parameter. The third data stream setconsists of the first data stream and the at least one third datastream.

As shown in FIG. 11, the present disclosure provides in some embodimentsa device for processing an adaptive media service at a decoder, whichincludes: a reception module configured to receive a second data streamselected by a transmitter in accordance with a condition of a receiver,the second data stream including a first section for transmitting secondimage encoding data and a second section for transmitting a targetoptimization parameter; a parsing module configured to parse the seconddata stream so as to acquire the second image encoding data carried inthe first section and the target optimization parameter carried in thesecond section; a decoding module configured to decode the second imageencoding data so as to obtain a second image sequence, image quality ofwhich is lower than that of an original first image sequence; and aquality improvement module configured to perform quality improvement onthe second image sequence in accordance with the target optimizationparameter, so as to obtain a third image sequence, image quality ofwhich is higher than that of the second image sequence.

In one possible embodiment, the device further includes a secondtransmission module configured to transmit the condition of the receiverto the transmitter.

In one possible embodiment, the target optimization parameter is anoptimization parameter selected from at least two available optimizationparameters in_param so as to provide MLM(LR,in_param) and the firstimage sequence with a maximum similarity, where LR represents the secondimage sequence, and MLM(LR,in_param) represents an image sequenceobtained through the quality improvement on the LR in accordance withthe available optimization parameter in_param.

In other words, the target optimization parameter may be determined inaccordance with such a condition that a first similarity is greater thanany second similarity. The first similarity is that between the thirdimage sequence and the first image sequence, and the second similarityis that between an image sequence obtained through quality improvementon the second image sequence in accordance with the other optimizationparameter in the available optimization parameters different from thetarget optimization parameter and the first image sequence.

In one possible embodiment, the device further includes a thirdtransmission module configured to transmit to the transmitter a messageindicating that the receiver is capable of parsing the targetoptimization parameter and performing the quality improvement inaccordance with the target optimization parameter, so as to enable thetransmitter to generate the second data stream and perform adaptiveselection from a set including the second data stream.

In one possible embodiment, the device further includes: a storagemodule configured to store the parsed target optimization parameter; andan extraction module configured to extract, prior to receiving a newtarget optimization parameter, the stored target optimization parameterfor the quality improvement module.

Of course, in order to reduce the data transmission volume caused by thetarget optimization parameter, in the embodiments of the presentdisclosure, the target optimization parameter carried in the secondsection is a compressed target optimization parameter, and the targetoptimization parameter is acquired by the parsing module throughdecompression.

The present disclosure further provides in some embodiments an encoderincluding the above-mentioned device for processing an adaptive mediaservice at an encoder.

The present disclosure further provides in some embodiments a decoderincluding the above-mentioned device for processing an adaptive mediaservice at a decoder.

A service system will be described hereinafter by taking the off-lineservice and the real-time service as examples.

As shown in FIG. 12, which is a schematic view showing a service systemfor the off-line service at the encoder according to one embodiment ofthe present disclosure, an image sequence (including one or more images)is inputted to an encoding module, and a bit stream having a data volumesmaller than or equal to that of the image sequence is outputted fromthe encoding module.

An operation procedure at the encoder will be described as follows inconjunction with FIG. 12.

A plurality of compressed bit streams (N+1 bit streams in FIG. 12) maybe inputted to the first selection and transmission modules. Then, thefirst selection and transmission modules may select and transmit one ofthe N+1 bit streams in accordance with a network condition and/or aplayback condition of a user terminal.

A first bit stream inputted to the first selection and transmissionmodules may be obtained by compressing an original image sequence HRthrough the encoder, and the other N bit streams may be obtained byperforming the other additional processing (e.g., resolution reduction)on the image sequence HR.

A processing procedure for each bit stream will be described as follows.

At first, the quality reduction (e.g., the resolution reduction) may beperformed on the image sequence HR by the quality reduction module usingsuch algorithms as High Efficiency Video Coding (HEVC), Automatic VolumeControl (AVC), Motion Picture Experts Group-2 (MPEG-2) or JointPhotographic Experts Group (JPEG) 2000, so as to acquire an imagesequence LR_m.

For the image sequence LR_m, the number of pixels and/or the number ofbits for each pixel are smaller than those of the image sequence HR. Forexample, the image sequence HR may include a red-green-blue (RGB) imagehaving a resolution of 1920*1080, with 24 bits for each pixel, while theimage sequence LR_m may include an image having a resolution of 960*540,with 8 bits for each pixel.

Next, the image sequence LR_m may be compressed via the encoding module,so as to acquire an original bit stream. For example, a 7500 YUV 4:2:0image having a resolution of 1920*1080 may be encoded by the encodingmodule into a 2 Mbps HEVC stream.

Next, a target optimization parameter corresponding to the imagesequence LR_m may be calculated by the first parameter determinationmodule using the above-mentioned methods.

Then, the target optimization parameter corresponding to the imagesequence LR_m may be compressed by the compression module. For example,a 1024-bit target optimization parameter that meets an Institute ofElectrical and Electronic Engineers (IEEE) Standard for Floating PointArithmetic may be compressed into a 5 KB bit stream using an LZ77algorithm and Huffman code.

Finally, the original bit stream and the compressed target optimizationparameter may be combined by the combination module, so as to acquirethe bit stream to be inputted to the first selection and transmissionmodules.

For example, an MPEG-TS stream including 2 Mbps HEVC video stream and a64 kbps AAC audio stream may be inputted to the encoding module, and thetarget optimization parameter may also be compressed into a bit stream.The two bit streams may then be combined into a final bit stream.

The first selection and transmission modules may receive from thereceiver the network condition and/or the playback condition, switchbetween different bit streams in an ascending order of the bit rates,and select the bit stream suitable for the receiver.

In different cases, these bit streams may be grouped into the followingsets: a data stream set A consisting of the first data stream, the atleast one second data stream and the at least one third data stream, adata stream set B consisting of the first data stream and the at leastone second data stream, and a data stream set C consisting of the firstdata stream and the at least one third data stream.

For example, the data stream set A may include a data stream (20 Mbps)having a resolution of 4K, transmitted at a speed of 30 fps (frames persecond) and encoded using an HEVC algorithm, a data stream (10 Mbps)having a resolution of 1080P, transmitted at a speed of 30 fps, encodedusing the HEVC algorithm and carrying an image sequence Param_1 (500KB), a data stream (10 Mbps) having a resolution of 1080P, transmittedat a speed of 30 fps, encoded using the HEVC algorithm and not carryingthe image sequence Param_1, a data stream (5 Mbps) having a resolutionof 720P, transmitted at a speed of 25 fps, encoded using the HEVCalgorithm and carrying an image sequence Param_2 (200 KB), a data stream(5 Mbps) having a resolution of 720P, transmitted at a speed of 25 fps,encoded using the HEVC algorithm and not carrying the image sequenceParam_2, a data stream (1 Mbps) having a resolution of 576P, transmittedat a speed of 25 fps, encoded using the HEVC algorithm and carrying animage sequence Param_3 (100 KB), and a data stream (1 Mbps) having aresolution of 576P, transmitted at a speed of 25 fps, encoded using theHEVC algorithm and not carrying the image sequence Param_3.

FIG. 13 is a schematic view showing a service system for the real-timeservice at the encoder according to one embodiment of the presentdisclosure. The service system in FIG. 13 differs from that in FIG. 12mainly in that the target optimization parameter is acquired indifferent modes.

The second parameter determination module needs to determine the imagetypes and the resolutions of the image sequences HR and LR_m, while thethird parameter determination module may call the correspondencerecorded in the data base, so as to determine the target optimizationparameter uniquely corresponding to the image type and the resolutionsof the image sequences HR and LR_m.

An image having a resolution of 1920*1080 may, for example, be recordedas an array of integers, i.e., [1080, 1920].

In the embodiments of the present disclosure, each combination includingan image type and a resolution of the image obtained after the qualityreduction may be assigned with a unique identity (ID), and the targetoptimization parameter corresponding to each ID may be stored in thedatabase.

Hence, after the image type and the resolution of the image obtainedafter the quality reduction have been determined, it is able to acquirethe ID, and then acquire the corresponding target optimization parameterin accordance with the ID.

As shown in FIG. 14, the present disclosure further provides in someembodiments a service system at a decoder, which includes the followingmodules.

The reception module, also called as a flow client device, may be usedto achieve the following two functions, i.e., a reception function forreceiving the compressed bit stream from an opposite end (thetransmitter), and a feedback function for collecting the networkcondition and/or playback condition of a local end, and transmittingthem to a server so as to enable the server to select a suitable bitstream in accordance with them.

In a conventional design, the reception module may be configured to:parse a manifest file or a file head so as to determine an availablemedia file and bit rate information; set and manage at least one sourcecache; request and download a content fragment to the source cache;process a media event; and etc.

The functions of the reception module are known in the art, and thuswill not be particularly defined herein.

Of course, in order to enable the opposite end to prepare for the datastream specifically, in the embodiments of the present disclosure, thefeedback function may also include transmitting to the serverinformation about whether or not the local end is capable of parsing andusing the target optimization parameter to perform the qualityimprovement, and any other relevant information (e.g., information abouta current version).

The reception module may be, for example, a conventional MPEG-DASHclient device.

After the reception module has received the data stream, the data streammay be separated by a separation module into the first section carryingthe second image encoding data and the second section carrying thetarget optimization parameter, and then the two sections may betransmitted to the decoding module and the parsing module for thesubsequent processings.

The decoding module corresponds to the encoding module, and the parsingmodule corresponds to the compression module, and thus they will not beparticularly defined herein.

In the case that the parsing module has parsed the target optimizationparameter, it is also required to update the original parameter in aregister, so as to facilitate the subsequent use.

Finally, the quality improvement module may perform the qualityimprovement on the decoded image sequence in accordance with theobtained target optimization parameter.

Of course, in the case of no target optimization parameter in thecurrent data stream, the quality improvement parameter may be performedin accordance with the target optimization parameter pre-stored in theregister.

The functional members described in the specification are referred to asmodules, so as to emphasize the independence of the implementation in amore particular manner.

According to the embodiments of the present disclosure, the modules maybe implemented by software, so as to be executed by various processors.For example, an identified, executable code module may comprise one ormore physical or logical blocks including computer instructions, and themodule can be constructed as an image, a process or a function. Even so,the executable codes of the identified modules are unnecessary to bephysically located together, but may comprise different instructionsstored in different locations. When these instructions are logicallycombined together, they form the modules and achieve the prescribedpurposes of the modules.

Actually, the executable code module may be a single instruction or aplurality of instructions, and can even be distributed at different codesegments, in different programs, or across a plurality of memorydevices. Also, operational data may be identified in the modules,implemented in any appropriate form, and organized in any data structureof an appropriate type. The operational data may be collected as asingle data set, or distributed at different locations (includingdifferent memory devices), and may be at least partially present in asystem or network merely as an electronic signal.

When the modules can be implemented by software, considering the currenthardware level, a person skilled in the art can build a correspondinghardware circuit to achieve the corresponding function if taking noaccount of the cost. The hardware circuit comprises a conventionalvery-large-scale integration (VLSI) circuit, a gate array, an existingsemiconductor such as a logic chip and a transistor, or other discretecomponents. The modules may further be implemented by a programmablehardware device, such as a field-programmable gate array, a programmablearray logic device and a programmable logic device.

The above are merely the preferred embodiments of the presentdisclosure, but the present disclosure is not limited thereto.Obviously, a person skilled in the art may make further modificationsand improvements without departing from the spirit of the presentdisclosure, and these modifications and improvements shall also fallwithin the scope of the present disclosure.

What is claimed is:
 1. A method for processing an adaptive media serviceat an encoder, comprising: a first acquisition step of acquiring a firstdata stream including first image encoding data obtained by encoding afirst image sequence, so as to enable a receiver to acquire the firstimage sequence in accordance with the first image encoding data, whereinthe first image sequence is an original image sequence; a secondacquisition step of acquiring at least one second data stream, differentsecond data streams having different image quality and each second datasteam including second image encoding data obtained by encoding a secondimage sequence and a target optimization parameter corresponding to thesecond image encoding data, so as to enable the receiver to decode thesecond image encoding data to obtain the second image sequence, andperform quality improvement on the second image sequence in accordancewith the target optimization parameter to obtain a third image sequence,the target optimization parameter being obtained in accordance with thefirst image sequence and the second image sequence, an identical contentbeing recorded in the first image sequence, the second image sequence,and the third image sequence, image quality of each second imagesequence being lower than those of the first image sequence and thethird image sequence; a first selection step of selecting one datastream from a first data stream set in accordance with a condition ofthe receiver, the first data stream set at least including the firstdata stream and the at least one second data stream; and a firsttransmission step of transmitting the selected data stream to thereceiver; wherein the second acquisition step comprises: a qualityreduction step of performing quality reduction on the first imagesequence so as to obtain the at least one second image sequence; anencoding step of encoding each of second image sequences so as to obtainthe second image encoding data corresponding to each of the second imagesequences in a one-to-one manner; a second parameter determination stepof determining an image type of the first image sequence and aquality-reduction level corresponding to each of the second imagesequences in a one-to-one manner; a third parameter determination stepof determining the target optimization parameter corresponding to eachof the second image sequences in accordance with a pre-storedcorrespondence among the quality-reduction level, the image type, andthe target optimization parameter; and a combining step of combiningeach of the second image encoding data and the corresponding targetoptimization parameter so as to obtain the at least one second datastream, wherein the target optimization parameter is an optimizationparameter selected from at least two available optimization parametersin_param so as to provide MLM(LR,in_param) and the first image sequencewith a maximum similarity, where LR represents the second imagesequence, and MLM(LR,in_param) represents an image sequence obtainedthrough the quality improvement on the LR in accordance with theselected optimization parameter in_param; wherein the method furthercomprises a compression step of compressing the target optimizationparameter corresponding to each of the second image sequences, whereinthe combining step further comprises combining the second image encodingdata and the corresponding compressed target optimization parameter; andwherein the second data stream comprises a meta data section and anattachment support section, and the target optimization parameter isstored in the meta data section or the attachment support section. 2.The method according to claim 1, further comprising: a third acquisitionstep of acquiring at least one third data stream corresponding to the atleast one second data stream, each third data stream including thecorresponding second image encoding data rather than the targetoptimization parameter; and a first switching step of, in the case thata data volume of the target optimization parameter is greater than apredetermined threshold, entering the first selection step and otherwiseentering a second selection step, wherein the second selection stepcomprises selecting one data stream from a second data stream set, thesecond data stream set consists of the first data stream and the atleast one second data stream, and the first data stream set consists ofthe first data stream, the at least one second data stream and the atleast one third data stream.
 3. The method according to claim 1, furthercomprising a third acquisition step of acquiring at least one third datastream corresponding to the at least one second data stream, each thirddata stream including the corresponding second image encoding datarather than the target optimization parameter, wherein the first datastream set further comprises the at least one third data stream.
 4. Themethod according to claim 1, further comprising: a third acquisitionstep of acquiring at least one third data stream corresponding to the atleast one second data stream, each third data stream including thecorresponding second image encoding data rather than the targetoptimization parameter; and a second switching step of, in the case thatthe receiver is capable of parsing the target optimization parameter andperforming the quality improvement in accordance with the targetoptimization parameter, entering the first selection step, and otherwiseentering a third selection step of selecting one data stream from athird data stream set consisting of the first data stream and the atleast one third data stream, wherein the first data stream set consistsof the first data stream and the at least one second data stream.
 5. Themethod according to claim 1, further comprising a second determinationstep of determining, prior to acquiring the at least one second datastream, whether or not the target optimization parameter needs to beupdated, so as to acquire a determination result, entering the secondacquisition step in the case that the determination result indicatesthat the target optimization parameter needs to be updated, andotherwise entering a replacement step, wherein the replacement stepcomprises acquiring at least one third data stream corresponding to theat least one second data stream, selecting one data stream from a thirddata stream set, and entering the first transmission step; each thirddata stream comprises the corresponding second image encoding datarather than the target optimization parameter; and the third data streamset consists of the first data stream and the at least one third datastream.
 6. The method according to claim 1, wherein the image type ofthe first image sequence includes cartoon image type or naturallandscape image type.
 7. A method for processing an adaptive mediaservice at an encoder, comprising: a first acquisition step of acquiringa first data stream including first image encoding data obtained byencoding a first image sequence, so as to enable a receiver to acquirethe first image sequence in accordance with the first image encodingdata, wherein the first image sequence is an original image sequence; asecond acquisition step of acquiring at least one second data stream,different second data streams having different image quality and eachsecond data steam including second image encoding data obtained byencoding a second image sequence and a target optimization parametercorresponding to the second image encoding data, so as to enable thereceiver to decode the second image encoding data to obtain the secondimage sequence, and perform quality improvement on the second imagesequence in accordance with the target optimization parameter to obtaina third image sequence, the target optimization parameter being obtainedin accordance with the first image sequence and the second imagesequence, an identical content being recorded in the first imagesequence, the second image sequence, and the third image sequence, imagequality of each second image sequence being lower than those of thefirst image sequence and the third image sequence; a first selectionstep of selecting one data stream from a first data stream set inaccordance with a condition of the receiver, the first data stream setat least including the first data stream and the at least one seconddata stream; and a first transmission step of transmitting the selecteddata stream to the receiver; wherein the second acquisition stepcomprises: a quality reduction step of performing quality reduction onthe first image sequence so as to obtain the at least one second imagesequence; an encoding step of encoding each of second image sequences soas to obtain the second image encoding data corresponding to each of thesecond image sequences in a one-to-one manner; a first determinationstep of determining a service type of the adaptive media service; afourth parameter determination step of, in the case that the servicetype of the adaptive media service is a real-time service, acquiring thetarget optimization parameter corresponding to each of the second imagesequences in accordance with a pre-stored correspondence among aquality-reduction level, an image type, and the target optimizationparameter, and otherwise, calculating the target optimization parametercorresponding to each of the second image sequences; and a combiningstep of combining each of the second image encoding data and thecorresponding target optimization parameter, so as to obtain the atleast one second data stream, wherein the target optimization parameteris an optimization parameter selected from at least two availableoptimization parameters in_param so as to provide MLM(LR,in_param) andthe first image sequence with a maximum similarity, where LR representsthe second image sequence, and MLM(LR,in_param) represents an imagesequence obtained through the quality improvement on the LR inaccordance with the selected optimization parameter in_param; whereinthe method further comprises a compression step of compressing thetarget optimization parameter corresponding to each of the second imagesequences, wherein the combining step further comprises combining thesecond image encoding data and the corresponding compressed targetoptimization parameter; and wherein the second data stream comprises ameta data section and an attachment support section, and the targetoptimization parameter is stored in the meta data section or theattachment support section.
 8. The method according to claim 7, whereinthe image type of the first image sequence includes cartoon image typeor natural landscape image type.
 9. A device for processing an adaptivemedia service at an encoder, comprising: a first acquisition circuitconfigured to acquire a first data stream including first image encodingdata obtained by encoding a first image sequence, so as to enable areceiver to acquire the first image sequence in accordance with thefirst image encoding data, wherein the first image sequence is anoriginal image sequence; a second acquisition circuit configured toacquire at least one second data stream, different second data streamshaving different image quality and each second data steam includingsecond image encoding data obtained by encoding a second image sequenceand a target optimization parameter corresponding to the second imageencoding data, so as to enable the receiver to decode the second imageencoding data to obtain the second image sequence, and perform qualityimprovement on the second image sequence in accordance with the targetoptimization parameter to obtain a third image sequence, the targetoptimization parameter being obtained in accordance with the first imagesequence and the second image sequence, an identical content beingrecorded in the first image sequence, the second image sequence, and thethird image sequence, image quality of each second image sequence beinglower than those of the first image sequence and the third imagesequence; a first selection circuit configured to select one data streamfrom a first data stream set in accordance with a condition of thereceiver, the first data stream set at least including the first datastream and the at least one second data stream; and a first transmissioncircuit configured to transmit the selected data stream to the receiver;wherein the second acquisition circuit is further configured to: performquality reduction on the first image sequence so as to obtain the atleast one second image sequence; encode each of second image sequencesso as to obtain the second image encoding data corresponding to each ofthe second image sequences in a one-to-one manner; determine an imagetype of the first image sequence and a quality-reduction levelcorresponding to each of the second image sequences in a one-to-onemanner; determine the target optimization parameter corresponding toeach of the second image sequences in accordance with a pre-storedcorrespondence among the quality-reduction level, the image type, andthe target optimization parameter; and combine each of the second imageencoding data and the corresponding target optimization parameter so asto obtain the at least one second data stream, wherein the targetoptimization parameter is an optimization parameter selected from atleast two available optimization parameters in_param so as to provideMLM(LR,in_param) and the first image sequence with a maximum similarity,where LR represents the second image sequence, and MLM(LR,in_param)represents an image sequence obtained through the quality improvement onthe LR in accordance with the selected optimization parameter in_param;wherein the device is further configured to compress the targetoptimization parameter corresponding to each of the second imagesequences, wherein the device is further configured to combine thesecond image encoding data and the corresponding compressed targetoptimization parameter; and wherein the second data stream comprises ameta data section and an attachment support section, and the targetoptimization parameter is stored in the meta data section or theattachment support section.
 10. The device according to claim 9, whereinthe image type of the first image sequence includes cartoon image typeor natural landscape image type.
 11. An encoder, comprising the deviceaccording to claim
 9. 12. A device for processing an adaptive mediaservice at an encoder, comprising: a first acquisition circuitconfigured to acquire a first data stream including first image encodingdata obtained by encoding a first image sequence, so as to enable areceiver to acquire the first image sequence in accordance with thefirst image encoding data, wherein the first image sequence is anoriginal image sequence; a second acquisition circuit configured toacquire at least one second data stream, different second data streamshaving different image quality and each second data steam includingsecond image encoding data obtained by encoding a second image sequenceand a target optimization parameter corresponding to the second imageencoding data, so as to enable the receiver to decode the second imageencoding data to obtain the second image sequence, and perform qualityimprovement on the second image sequence in accordance with the targetoptimization parameter to obtain a third image sequence, the targetoptimization parameter being obtained in accordance with the first imagesequence and the second image sequence, an identical content beingrecorded in the first image sequence, the second image sequence, and thethird image sequence, image quality of each second image sequence beinglower than those of the first image sequence and the third imagesequence; a first selection circuit configured to select one data streamfrom a first data stream set in accordance with a condition of thereceiver, the first data stream set at least including the first datastream and the at least one second data stream; and a first transmissioncircuit configured to transmit the selected data stream to the receiver,wherein the second acquisition circuit is further configured to: performquality reduction on the first image sequence so as to obtain the atleast one second image sequence; encode each of second image sequencesso as to obtain the second image encoding data corresponding to each ofthe second image sequences in a one-to-one manner; determine a servicetype of the adaptive media service; acquire, in the case that theservice type of the adaptive media service is a real-time service, thetarget optimization parameter corresponding to each of the second imagesequences in accordance with a pre-stored correspondence among aquality-reduction level, an image type, and the target optimizationparameter, and otherwise, calculate the target optimization parametercorresponding to each of the second image sequences; and combine each ofthe second image encoding data and the corresponding target optimizationparameter, so as to obtain the at least one second data stream, whereinthe target optimization parameter is an optimization parameter selectedfrom at least two available optimization parameters in_param so as toprovide MLM(LR,in_param) and the first image sequence with a maximumsimilarity, where LR represents the second image sequence, andMLM(LR,in_param) represents an image sequence obtained through thequality improvement on the LR in accordance with the selectedoptimization parameter in_param; wherein the device is furtherconfigured to compress the target optimization parameter correspondingto each of the second image sequences, wherein the device is furtherconfigured to combine the second image encoding data and thecorresponding compressed target optimization parameter; and wherein thesecond data stream comprises a meta data section and an attachmentsupport section, and the target optimization parameter is stored in themeta data section or the attachment support section.
 13. The deviceaccording to claim 12, wherein the image type of the first imagesequence includes cartoon image type or natural landscape image type.14. An encoder, comprising the device according to claim 12.